3 posts
The single decision that most human-in-the-loop projects get wrong. If your eval labels were seeded by the model's own predictions, every F1 number you ever report is biased toward the model. The fix is cheap on day one, expensive on day forty.
After every batch of human-corrected episodes gets exported from Label Studio, the temptation is to retrain immediately. The reasons not to, and a cheap cadence trigger that actually fires when retraining will help.
When you fine-tune model v3, you need to be able to answer "which exported corrections went into it". Snapshotting the whole training set is the obvious answer and the wrong one. Track the inputs and the derivation; the training set is a function of them.